Developing an Evaluation Framework for a Custom-designed Course Management System

This study evaluated a custom-designed course management system in use at a mid-sized public institution in the Pacific Northwest region of the United States. While focusing primarily on gathering information relevant to the continued development of the course management system, this study sought to identify the relationship among user characteristics, usability factors, and acceptance of the course management system. Additionally, this study sought to identify common usability problems with specific course features encountered by the users. Participants in this study were 308 undergraduate and graduate students who were currently using the online course management system to take an Internet-enhanced or fully Internetdelivered course. The data collected via an online survey instrument indicated that three usability factors, ease of navigation, ease of learning, and visual perception were significant predictors of acceptance of the course management system. The usability of specific course management system features was examined using both Likert-scale items and content analysis of open-ended survey questions. The content analysis of the open-ended items revealed several specific problems that were commonly encountered while using specific course features. Based on these findings, several recommendations are made toward the improvement of the customdesigned course management system and also toward the improvement of the survey instrument for future use.